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Abstract 



Forty faculty volunteers from 30 departments were randomly assigned to one of 
three experimental conditions: ^'full process," with teaching performance data 
collection, feedback, diagnosis, applied instructional improvement strategies 
and data re-collection; "diagnostic," which excluded the applied improvement 
strategies; and *'data collection only." Full process and diagnostic condition 
faculty were assisted throughout by graduate student "teaching improvement 
specialists." Late semester findings were that full process instructors con- 
sidered the process to be effective, worth their time and effort, and appro- 
priate for their peers. There were no across treatment differences in faculty 
attitudes or self-ratings of teaching skill improvement, need for improvement, 
or overall teaching competence. Nor were there consistent across treatment 
differences in their students' attitudes or ratings of instructor teaching 
skill and improvement. 



AN EVALUATION OF A TEACHING IMPROVEMENT PROCESS FOR UNIVERSITY FACULTY 



There are more than 250 college and university instructional im- 
provement centers and programs in this country (Gaff, 1975). Their 
intellectual traditions, assessments of the most critical teaching and 
learning problems, and prescriptions vary, but all of those identified 
by Gaff are attempting to improve instruction through the in-service 
education of college teachers. Almost all offer at least some one-on- 
one consultation service to faculty members with course design or class- 
room teaching problems. Many offer consultation which is intended to 
help instructors improve their performance of underdeveloped skills or 
to broaden often limited teaching skill repertoires. 

The Clinic to Improve University Teaching (Clinic)* has developed 
a systematic and individualized teaching improvement process which it 
believes has broad applicability to other instructional improvement 
programs. That process has several features which make it unique. 
First, its diagnostic and improvement procedures center upon some 
twenty teaching skills which are related to students' and instructors* 
perceptions of what constitutes effective teaching across a broad range 
of instructional situations. Second, the Clinic's diagnostic procedures 
include data collection techniques designed to obtain and integrate 
information from a variety of sources. These include student and in- 
structor questionnaires, classroom observation and videotaping, 



*The Clinic is supported by a grant from the W.K. Kellogg Foundation. 
Both authors were on the Clinic staff when the evaluation study described 
herein was conducted, and were ably assisted by B. Erickson, B. Hov/ard, 
M. Jackson, L. Klemer-Hruska , D. McCarthy, and Wright. 
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interviews, and the examination of course materials. Third, im- 
provement goals are negotiated and worked toward by selecting or 
designing and using any of a number of teaching improvement strategy 
alternatives. And, finally, faculty members are provided with con- 
tinuing individualized assistance throughout the process by people- 
thoroughly trained as ^*teaching improvement specialists." The process 
is designed to help an instructor identify significant strengths and 
problems within some course and to acquire or develop teaching skills 
which will increase his or her effectiveness within that course. 

Unfortunately, evidence about the impact and the success or fail- 
ure of the consultation strategy used by the Clinic has not been avail- 
able. Nor have other teaching improvement programs (excepting, perhaps, 
instructional design programs) produced substantial evidence about 
their effectiveness. This is particularly unfortunate in a time when 
interest in "teacher effectiveness" and its improvement is high even 
while resources are shrinking. Such evidence is critical now for the 
many institutions considering the development, elimination or modifi- 
cation of their own instructional improvement programs. 

We believe that a major reason for the scarcity of evaluative in- 
formation has been a paraly/inc; preoccupr^tion vn'th v/hat many consider 
to be the "ultimate" questions: Are teaching skills, behaviors and 
attitudes amenable to significant, quarti f icible <ir»c! Icsling cfjenrjt? 
Ar.d, even if so, vnll studerts either learn more or more efficiently 
as a result of such changes? Important questions, yes, but not the 
only significant ones and probably not ttie best or«es wilh v-hich to 
begin. For programs directed toward improving university teaching to 
have any chance for success with reference to those "ultimate" questions, 
it would seem that they must be perceived as practical, effective and 
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attractive by consumers. Patricia Cross (1976, p. 3) is right: "... 
the reality is that instructional improvement is dependent upon faculty 
cooperation." This evaluation study, then, deals with faculty satis- 
faction with the Clinic's teaching improvement process, with student 
and faculty perceptions of qualitative changes in instructional per- 
formance, and with student and faculty attitudes toward themselves, 
their courses, and teaching. 

Method 

Subjects. The subjects were 40 volunteer faculty (28 males; 12 
females) from 30 academic departments of a large New England university. 
Six were staff associates or instructors, 17 were assistant professors, 
11 were associate professors, and six were full professors. Class sizes 
ranged from eight to over 200 students, with most falling into the 15 to 
60 range. The subjects were probably not representative of the faculty 
of the university, but were similar in age, sex, rank, discipline and 
class size to faculty members who had volunteered to work with the Clinic 
in the past. 

The teaching^ imprqyement P!;;^ces$. The Clinic's improvement process 
is undertaken by instructors within the context of one of their courses 
over a semester. In collaboration with a graduate student trained as a 
"teaching improvement specialist," they work through four stages: 1) dat 
collection; 2) feedback and diagnosis; 3) applied teaching improvement 
strategies; and 4) data re-collection and review. During the first stage 
data regarding the instructor's teaching goals and objectives, procedures 
and performance of certain teaching skills are collected through an in- 
terview, classroom observation and videotape, and a student and instruc- 
tor teaching questionnaire. During the second stage, the instructor and 
improvement specialist review the data, jointly identify significant 
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strengths and problems, and establish reasonable improvement goals. 
Various teaching improvement strategies are devised and used during the 
third and longest stage. And^ finally, additional data about perceived 
changes in the quality of teaching performance are collected. 

Treatment conditions . Subjects were randomly assigned to one of 
three treatments: a "full process*' condition which entailed going 
through the entire Clinic process as just described; a "diagnostic" 
condition which excluded the development and application of teaching 
•improvement strategies (although possible improvement strategies were 
offered orally after diagnosis); and a "data collection only" con- 
dition which included only the first and the last stages of the teach- 
ing improvement process, with no feedback until the study was concluded. 
Subjects were then randomly assigned to five experienced teaching im- 
provement specialists, all doctoral level students in Education. 

Diagnostic instruments . The Teaching Analysis by Students ( TABS ) 
questionnaire was developed by the Clinic in 1974- for use as a diagnostic 
instrument within their teaching improvement process. It includes 38 
teaching skill statements which are responded to on a four-point "amount- 
of-improvement-needed" scale. An additional item asks for a general 
rating of the course (TABS is reproduced in Bergquist, Phillips and 
Quehl , 1975). The TABS was administered to each instructor's class 
during the fourth week of the semester. At that time, each instructor 
also rated his or her own teaching performance on the TABS . These 
results were used to check on the early semester comparability of the 
three groups of subjects and as covariates for post measures. 

Late semester measures . Near the end of the semester each instruc- 
tor's students completed the three-section Student Questionnaire . Section 
I, different for^each instructor, asked students to indicate on a five 
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point scale (much better to much worse) the extent of qualitative change 
they had observed over the preceding six weeks in their instructor's 
performance of the ten teaching skills which they had rated on the 
earlier TABS as most needing improvement. Section II included 35 teach- 
ing skill items with the scale from the TABS, and an item for an overall 
rating of the instructor* Section III was a three concept ("As a student 
in this course I feel. . .," "As a teacher, this instructor seems. . 
and "This course seems. . .") semantic-differential with seven-point 
bipolar adjective scales (see Figure 1). 

At the same time, instructors completed the Instructor Questionnaire 
which, excepting the semantic-differential, was identical for each in- 
structor to the Student Questionnaire completed by his or her students. 
The final section was also a semantic-differential, with similar adjec- 
tive scales (see Figure 1) but different concepts ("In general, as a 
teacher I am. . .," "As a class, these students seem. . .," and "For me, 
this course is. . ."). 



Insert Figure 1 about here 



Finally, after finishing the diagnostic stage of the process, full 
process and diagnostic condition subjects completed the 25 item Evaluation 
of Teaching Clinic: Part I . Full process condition subjects completed 
an additional 14 item Part II at the conclusion of the process. Both 
parts included a mix of forced-choice (mostly strongly agree to strongly 
disagree) and free-response items designed to solicit reactions to their 
improvement specialist and the elements of the improvement process. 

Results 

Student data . A comparison of class means across the three 
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treatment conditions revealed statistically significant group differ- 
ences on the early semester TABS results. Consequently, subsequent 
analyses were restricted to data from students who completed both early 
and late semester measures so that analysis of covariance technique? 
could be used to adj'ust final class means for initial differences. 
(Comparisons of the scores of students who completed both instruments 
with those of students who completed only one or the other revealed 
no significant differences.) 

Using class means as the unit of analysis, one-way analyses of 
covariance were carried out on the following late semester Student Ques- 
tionnaire scores: 1) ratings of qualitative changes in the performance 
of the ten teaching skills which, according to TABS results, were most 
in need of improvement for each instructor; 2) ratings of the degree of 
improvement still needed on the 35 teaching skill items carried over 
from the TABS; and 3) overall instructor ratings. The covariates were, 
respectively: 1) the average amount-of-improvement-needed scores on 
the corresponding ten TABS items; 2) the average amount-of-improvement- 
needed scores on the corresponding 35 TABS items; and 3) the initial 
overall course rating scores from the TABS. The latter were also used 
as covariates for the multivariate analysis of covariance on thf^ semantic- 
differential concept average scores. 

The Student Questionnaire results are summarized in Table 1. There 
were no differences across treatment conditions in student perceptions 
of improvement on the ten weakest teaching skills for each instructor 
(F=2.60, df=2/36, p<.10) or for overall instructor ratings (F=2.68, 
df=2/36, p<.10). There were, however, significant differences (F=3.30, 
df=2/36, p<.05) in final amount-of-improvement-needed ratings for the 
35 teaching skill items. (Post hoc comparisons with Scheffe's test 
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showed that full process condition adjusted mean scores were signify 
icantly higher than those of the diagnostic, but not data collection 
only, condition classes. There were no differences between diagnostic 
and data collection only condition scores.) With regard to attitude 
scores as measured by the semantic-differential, there were no differences 

Insert Table 1 about here 



across treatments in attitudes toward "self as student" or the course. 
But there were significant differences in attitudes toward the "instruc- 
tor as teacher." (Scheffe's post hoc test indicated that full process 
condition adjusted mean scores were significantly greater than those of 
data collection only, but not diagnostic, condition classes. There were 
no other differences.) 

Instructor data . As there were no significant across treatment 
condition differences on early semester instructor sel f -assessments , 
Instructor Questionnaire results were compared through analysis of var- 
iance. The results of these analyses are summarized in Table 2. They 
indicate that there were no differences across treatment conditions on 
late semester instructor self -assessments or in attitudes toward them- 
selves as teachers, their courses, or their students. 



Insert Table 2 about here 



Finally, instructors' responses to the Evaluation of Teaching 
Clinic questionnaire are summarized in Table 3. For all Likert scale 
items, an average of 93.6% of the responses were positive. Of particular 
interest, perhaps, are the strongly positive responses to the iterfiS'" having 
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to do with the utility of improvament strategies for helping to improve 
their teaching (#31), whether the entire process was useful and worth 
their time and effort (#36), and whether they would recoimiend the 
process to their peers (#37). 



Insert Table 3 about here 



Discussion 

This evaluation study did yield one unambiguous finding: facultr 
members who undertook the Clinic's entire teaching improvement proce^as 
were pleased with the experience. They agreed that it helped them to 
improve their teaching, that it was worth their time and effort, that 
they would recommend it to their peers, and that their graduate student 
teaching improvement specialists were competent and helpful. It is 
true that all participants were volunteers and that Gaff (1975, p. 168) 
has noted that it has been the experience of several programs that 

. . participants generally find what they come for." Nonetheless, 
client satisfaction is an important goal for the Clinic and for other 
teaching improvement programs. Furthermore, some may be surprised that 
graduate students can be a credible medium for providing faculty members 
with what they obviously consider to be a very useful service. 

Less clear is the significance of the lack of across treatment 
differences in late semester instructor self-ratings of improvement 
demonstrated (all felt that there had been improvements) or needed and 
of overall teaching competence- Nor were there differences in late 
semester attitudes toward themselves, their courses, or their students. 
Perhaps those results suggest that each of the faculty members who 
volunteered was indicating a willingness to change, and all may have 



been encouraged to improve by the knowledge that their teaching was 
being examined for evidence of change • That explanation suggests that 
the improvement strategies used may well have worked as reported on 
item #31 of the Evaluation of Tea ching Clinjc. But they may not have 
resulted in changes which were experienced as any more intense than 
were the adjustments made by the atypically motivated subjects in the 
other experimental groups. As for instructor attitudes, there is simply 
no evidence that the Clinic's process had any impact. 

Student responses to the Student Q uestionnai re are not very illu- 
minating either. Generally, there are no across treatment condition 
differences, except that full process adjusted mean scores are more 
positive than those of one of the two, but not both, other groups for 
two variables: average amount of improvement still needed on all of 
the skills, and attitudes toward their instructors as teachers. The 
significance of those results may have been clearer if the full process 
scores had been more positive than those of both other groups, or of 
the same group, for both measures. As it stands, there is no real 
evidence that student attitudes or perceptions of their instructors* 
teaching was affected by the treatment condition to which their in- 
structors were assigned. 

In sum, there is no compelling evidence from either instructor 
self-ratings or student ratings that their teaching changed any more or 
less than that of instructors in the other two treatment groups. In 
fact, instructors and students perceived positive changes in teaching 
performance in all three groups. What is clear about the Clinic's 
teaching improvement process from this evaluation study is that it 
produces consumer satisfaction with the process, the mediator, and 
the results. 



Some of the other results may have been less ambiguous if the 
evaluation design had been more precise. For example, we did not ask 
teaching improvement specialists to restrict their instructional im- 
provement prescriptions and efforts to any abbreviated list of skills. 
To do so may have meant that subjects/clients would have spent time 
working on less personally significant teaching problems. But to have 
limited their attention to five or ten teaching skills, and/or to have 
assigned subjects to groups after problem identification, may have 
allowed for more precise assessments of the consequences of working on 
or ignoring shared problems. 

This evaluation study, then, put aside for the moment any concern 
with the questions of whether teaching skills and behaviors are amenable 
to significant change and v;hether any such change would significantly 
affect student learning. It focused instead upon consumer satisfaction, 
perceptions of change, and some related attitudes. That faculty members 
were satisfied with the Clinic's teaching improvement process is obvious 
and encouraging. But the study failed to demonstrate that instructors 
who complete the Clinic's process will appear to themselves or their 
students to have improved any more than instructors in diagnostic or 
data collection only groups. Nor did we find any clear impact upon 
student or instructor attitudes toward themselves, their classes, or 
each other. We believe, however, that there is just enough ambiguity 
in the data to suggest that these findings may be more of a reflection 
upon the imprecision of the study's design and measures than of the 
ineffectual ity of the Clinic's teaching improvement process. Certainly 
we hope that the Clinic and other instructional improvement programs will 
continue and step up the pace of their efforts to assess the impacts of 
their programs—even if most of us still lack the skills or the resources 
for investigating program effects upon student learning. 
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Concepts 



Adjective Pairs 



As a student in this course I feel., 
As a teacher, this instructor seems, 

In general, as a teacher I am... 
As a class» these students seem... 



sel f-conf i dent/ not sel f -conf i dent 
irresponsible/ responsible 
unsuccessful/successful 
organized/disorganized 

passive/active 
not anxious/a vious 

uninformed/informed 

competent/ incompetent 

tense/not tense 

enthusiastic/not enthusiastic 
(about learning) 



This course seems. . . 
For me, this course is. 



rewardi ng/unrewa rdi ng 
stimulating/dull 

rel evant/i rrel evant 
unpl easant/pl easant 
impersonal /personal 
meaningful /meaningless 



Figure 1: Semantic-differential concepts and bipolar adjective pairs used 
in the third sections of the Student Questionnaire and the 
Instructor Questionnaire . 



Table 1 

Late Semester Student Questionnaire Results 



Measures 



Treatment Condition 



Full Process Diagnostic Data Only 









(N=13)b 


(N=13) 


(N=14) 




Perceived improvement on ten 
weaKesu Leacmng ski i is 


Adj. 
Obs. 


X 

I 
SD 


.22 


2.58 

2.61 
.16 


2.69 

2.70 
.15 


.10 


Average amount of improvement 
sl 1 1 1 Nccucu on JO Lcdcniny 
skill items^ 


Adj. 
Obs. 


I 

T 
SD 


. il.bb, - 

1.80 
.31 


(2.07)^ 

2.18 
.19 


2.01 

1.99 
.26 


.05 


Overall instructor ratingsS 


Adj. 


T 


1.95^ 


2.26 


2.32 


.10 




Obs. 


I 
SD 


1.85 
.61 


2.51 
.49 


2.18 
.65 




Semantic-differential 
concept scoresS 














Toward self as student 


Adj. 


X" 


5.23^ 


5.00 


5.09 


.26 




Obs. 


T 
SD 


5.28 
.42 


4.90 
.39 


5.14 
.32 




Toward instructor as 
teacher 


Adj. 
Obs. 


I 

I 
SD 


(6.06)^ 

6.11 
.33 


5.85 

5.74 
.31 


(5.64) 

5.70 
.39 


.005 


Toward course 


Adj. 


K 


5.53h 


5.31 


5.14 


-.10 




Obs. 


J 
SD 


5.67 
.59 


5.01 
.63 


5.29 
.78 





^Probability of differences across treatments being by chance 
'^Number of classes 

^Covariates were earlier scores on corresponding TABS items 

^Scores could range from one (most positive) to five 

^Scores could range from. one (most positive) to four 

"^Adjusted means within parentheses and in the same row differ significantly 
according to Scheffe's post hoc test 

9Covariates were overall course ratings an early semester TABS 

^Scores could range from one (most negative) to seven 
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Table 2 

Late Semester Instructor Questionnaire Results 



Treatment Condition 



Measures 




Full Process 


Diagnostic 


Control 




Perceived improvement on ten 


X 


2.32'^ 


2.40 


2.68 


<.10 


weakest teaching skills 


SD 


.42 


,..35 


.27 






12 


10 


11 




Average amount of improvemer * 






2.15 


2.03 


':.53 


stili needed on 35 teaching 




.43 


.31 


.37 


skill Items 

1^ lit 1 W ^» III «J 














12 


10. 


11 




Overall instructor rating 


x" 


2. 18^ 


? 44 




fin 




SD 


!58 


.50 


.64 






N 


11 


9 


10 




Semanti c-di f f erenti al 












concept scores 












Toward self as teacher 


X 


5.77^ 


5.72 


5.55 


• .89 




SD 


.60 


.58 








N 


12 


13 






Toward course 


X 


6.42^ 


5.86 


c 


.16 




3D 


.64 


.72 


1 . 






N 


12 


13 






Toward students 


T 


5.19^ 


4.45 


4.-.. 


. 12 




sp 


.79 


.77 


I.IJ 






N 


12 


13 


10 





^Probability of differences across treatments being by chance 
'^Scores could range from one (most positive) to ^ive 
^Number of instructors 



"Scores could range from one fpost positive) to rour 
^Scores coiuld range from (most negative) to seven 
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Table 3 



Summary of Instructor Responses to 
Evaluation of Teaching Clinic Questionnaire Items 



Itemsb 




Responses 


Positive^ 


Negative Undecided 


Part I: 


completed by full process and diagnostic 






condition subjects after diagnosis 





1. By the end of the initial interview, the 

Clinic's goals were quite clear to me. 26 - 0 0 

2. The procedures used within the Clinic's 
teaching improvement process were clearly 

explained to qb. 25 0 1 

3. Procedures and instruments for collecting 

data about my teaching were not made clear to me. 24 2 0 

4. I do not believe that the preliminary 
information about my course and my teaching 
methods that was gathered through the Cou^rsr 
Information Form, the initial inter ; ^w, sTid 
the course materials which I made a^a-larle 
gave the teaching improvement specia" :st (TIS) 
an adequate perspective from which tx assess 

data to be collected later. 23 2 1 

5. During the initial interview, I vsb:^ 
able to clarify my personal objective 

in working with tiiE Clinic. 23 2 1 

6. This initial slage did not give £ 
very clear idea of how much. of my t mae snc 
my class's time wauld be required b the 

Clinic's data collection procedures, 23 3 0 

7. The TIS failed to answer some o^ 

questions satisfactorily. " 25 1 0 

8. All in all, I am convinced that t:\e 
initial interview v/as well worth my t-'-rvB 

and effort. 25 10 



^Strongly agree or agree on positTive^ stated items; strongly disagree or 
disagree for negatively stated items. Rule -^or negative responses is just the 
reverse. 

'^Items 24, 25, 32 and 39 have beei-^ orrttted because of space limitations. 
Two of them (24 and 32) asked for ratir -is ^r^ the usefulness of the various data 
sources, and the other two asked for a^adftncnal comments or elaborations. 

18 
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Table 3 (continued) 

SuiTwary of Instructor Responses to 
E valuation of Teaching Clinic Questionnaire Items 



Responses 

^^^^^ Positive Negative Undecided 



9. The potential value of each type of data which 
was collected (observation, videotape, student 
IAB5» self-assessment, and prediction of student 
responses to IMS) was persuasively explained 

to me. 23 1 2 

10. The TIS*s observation and videotaping of 
my class was disruptive or otherwise poorly 

handled. 24 1 0 

11. I was not happy with the way the admin- 
istration of the TABS questionnaire to my 

class was handled. 25 1 0 

12. The procedures for completing the lAES.' 
self-assessment and prediction of student 

responses were explained clearly. 24 2 0 

13. Generally, I do not believe that the 
data which was collected wars sufficiently 
comprehensive to provide the basis for 
making decisions about my significant 

teaching strengths and weaknesses. 21 ^ 1 

14. I did not find the opportunity to preview 
the videotape and JABS results by myself useful. 

15. The computer printout of the TABS results 
was confusing or otherwise not an adequate 

display of the data. ?A I I 

16. The TIS appeared to be well prepared for 

our localization session. 26 0 0 

17. The TIS did a fine job of presenting and 
explaining the data which was collected about 

my teaching performance. 25 0 1 

18. Even with the TIS's assistance, I was over- 
whelmed by the amount and/or the variety of 

data which was made available to me. 23 2 1 

19. We were able to identify some significant 
teaching strengths which were indicated by the 

data. 25 0 1 



19 



25 1 0 
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Table 3 (continued) 



Summary of Instructor Responses to 
Evaluation of Tjg_aching CI i pic Questionnaire Items 



Items 




Responses 




Positive 


Negati ve~ 


Undecided 


20. The TIS and I did not agree on significant 
teaching problems which were substantiated by 
the available data. 


26 


0 


0 


?1 Thp TT^ w;) Q nni" aKIp iri ciinnocf" nK';ipi"ipa1 
C4 . 1 lie 1 lo Wu o 1 1 y L uL^ 1 v: lu oU^^coL piauLiUui 

and potentially worthwhile teaching improvement 
strategies for me to undertake. 


22 


0 


2 


22. I was not convinced that I should go on to 
the teaching improvement strategy phase of the 
instructional iinprovement process. 


23 


1 


1 


23. So far, I am convinced that my TIS is 
competent to work with me. 


26 


0 


0 


Part 11: completed only by full process condition 
subjects after entire process completed. 


26. I was impressed by the TIS's ability to 
suggest a wide variety of possible teaching 
improvement strategies. 


11 


1 


0 


27. The TIS was not adequately sensitive to 
my personal style, responsibilities, and specific 
instructional problems when suggesting possible 
teaching improvement strategies. 


11 


1 


0 


28. The teaching improvement strategies which 
I tried, or which were suggested to me, were 
not explained to my satisfaction. 


12 


0 


0 


29. The TIS was not of much help to me as I 
tried to iriiplement teaching improvement strategies 


. 12 


0 


0 


30. Adequate procedures for assessing the 
effectiveness of the teaching improvement 
strategies which I tried were not developed or 
used. 


11 


1 


0 


31. Overall, I believe thcit the teaching 
improvement strategies whic^ I tried helped me 
improve my teaching. 


12 


0 


0 
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Table 3 (continued) 



Summary of Instructor Responses to 
Evalua tion of Teachin g Clinic Questionnaire Items 



Responses 



Items 



Positive Negative Undecided 



33. I found it difficult to interpret the 
final questionnaire data as it was displayed 

and explained. 9 

34. I thought that efforts to collect data 
ah^^L \wa»iges^ rr. my teacning Qarformance were 
comprehensive and satisfactory. 10 

35. After the final data re-collection process, 
the TIS helped me identify current teaching 
strengths and any significant teaching problems. 12 

36. Overall, Z found the Clinic's teaching 
improvement process useful and worth my time 

and effort. 11 

37. I would not recommend the Clinic's 

teaching improvement process to my peers. 11 

38. I found my TIS competent and helpful. 12 



0 
0 
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